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Abstract 


Under a hybrid emporium instructional model, students primarily learn content and skills at 
their own pace through a computer-based platform; during class time, faculty serve more as 
tutors facilitating individual learning rather than as traditional lecturers. This study evaluates 
the adoption of this technology-centered instructional model in developmental math 
courses at public two- and four-year colleges in Tennessee. Using nine years of 
student enrollment and transcript data (2006—07 to 2015-16) provided by the Tennessee 
Board of Regents, this paper examines the effects of technology-centered instruction in 
developmental math courses on students’ course pass rates, persistence rates, and 
completion rates, compared to students’ outcomes in traditional lecture-based 
developmental math courses. While much of the prior research on the effects of 
technology-centered instruction has applied to students from a wide range of academic 
backgrounds, the primary objective of this paper is to discern whether technology- 
centered instruction is helpful (or harmful) for students who are academically 
underprepared for college. 


Using a difference-in-differences analytic model to exploit variations in institutions’ 
timelines in adopting the hybrid emporium model, I find that, for community college 
students, being assigned to a technology-centered developmental math course led to 
lower pass rates in their first college-level math course, fewer cumulative credits earned 
over time, and a lower likelihood of earning an associate degree within six years, as 
compared to students assigned to traditional developmental math courses. At four-year 
colleges, the adoption of this new instructional model resulted in a higher percentage of 
students passing their developmental math courses and thus spending fewer terms in 
developmental math. However, the pass rates of these students in their first college-level 
math courses were lower than those of students who were assigned to traditional 
developmental math courses. The magnitude of the effects varied by gender, age, and 
ACT math score. 
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1. Introduction 


Academic readiness remains one of the primary challenges to college student 
success. Colleges often place students who are not academically ready for college into 
developmental education courses to improve their math, reading, or writing skills and 
prepare them for college coursework. However, only half of all students who enroll in a 
developmental course successfully complete the course (Chen, 2016). Students who fail 
developmental math are less likely to ever earn a degree or credential compared to those 
who pass (Le, Rogers, & Santos, 2011). Over the past several years, an increasing number 
of states and institutions have received financial support from government and private 
sources to develop and assess alternative approaches to developmental education; others 
have begun to do so on their own. A 2018 national survey of state-level developmental 
education policies found that 20 states authorize the use of “innovative developmental 
education instructional methods and interventions” (Whinnery & Pompelia 2018). Newly 
redesigned developmental programs attempt to better target students’ academic needs, 
often through the use of learning technology such as self-directed learning labs, 
computer-based instructional models, and high-tech classrooms (Epper & Baker, 2009; 
Means, Toyama, Murphy, & Baki, 2013). The use of technology in postsecondary 
teaching has grown rapidly over time, particularly in developmental math courses (Allen 
& Seaman, 2007, 2010; Means, Bakia, & Murphy, 2014). 


Tennessee, the site for this study, is a national leader in the implementation of 
technology-driven instruction in developmental math. In the year 2000, more than 70 
percent of students enrolled at one of the state’s 13 community colleges were being placed 
into developmental math courses, along with more than 50 percent of students at one of 
the state’s six four-year public universities (Gray-Barnett, 2001). The drop/failure/ 
withdrawal rate in developmental math courses averaged 45 percent, compared to 26 
percent in college-level math courses. In 2005, in response to the high number of students 
in need of developmental math and the high course failure rates, the Tennessee Board of 
Regents (TBR), the governing body overseeing these 19 colleges, ' provided small grants 
to four institutions “to support technology-supported active learning strategies aimed at 
improving student learning outcomes, accelerating time to credit-bearing courses, and 


' For the years of this study, TBR was the governing body of the state’s 13 community colleges, six 
state universities, and 27 Tennessee colleges of applied technology (TCAT). Across these 46 public 
colleges, TBR served more than 200,000 students. In 2017, the six universities separated and are now 
governed by local boards. TBR, now known as The College System of Tennessee, currently enrolls 
nearly 120,000 students across the 13 community colleges and 27 TCATs. 


reducing instructional costs” as part of the state’s Developmental Studies Redesign 
Initiative (Crandall & Soares, 2015, p. 11). 


Each of the pilot colleges introduced a variation on a new instructional model at 
the time, known as the emporium model. The model was developed at Virginia Tech in 
1997, and the term “emporium model” was later broadly adopted by the National Center 
for Academic Transformation (NCAT) to describe a course in which traditional lectures 
are replaced with instructional software (National Center for Academic Transformation, 
2013). The term “hybrid” emporium model is often used to emphasize that instructors 
still play a key role in a course even when they do not function as traditional lecturers. 
Under the hybrid emporium model, students typically spend all their class time in a 
computer lab learning the course content online at their own pace and are allowed to 
continue their work at home or outside of class by logging into the course interface with 
their unique ID and password. During class, faculty float around the computer lab, serving 
more as tutors who deliver individualized instruction as opposed to traditional lecturers 
who address the entire class. Instead of grading homework assignments and assessments, 
faculty rely on the software to automatically grade student work and track student 
progress. This gives the faculty more time to answer students’ questions individually, 
both in the lab and through messages sent through the course site. The technology-based 
instruction used within this study was Pearson’s MyMathLab and Hawkes Learning 
System. Within the interface, the course content is divided into small blocks, or modules, 
which are taught using tutorials, short (3-5 minute) video lectures, sample problems, 
practice exercises, and quizzes and tests. When students fail a quiz or exam, they are 
allowed to complete the module again, with new sample problems. While the amount of 
class time spent in the computer lab can vary across institutions, the pedagogy under the 
hybrid emporium model is consistent: Whole-class instruction, including lectures, are 
eliminated and replaced with personalized instruction, mostly using interactive software. 


Following the adoption of these technology-based pilot courses in four Tennessee 
institutions from 2005 to 2008, observational research and descriptive summaries from 
developmental math redesigns at NCAT partner institutions suggested that the new model 
was highly successful (National Center for Academic Transformation, 2009; Twigg, 
2011). Comparing the pass rates of students before and after implementing the model 
showed that the percentage of students successfully completing a developmental math 
course increased 51 percent on average across the NCAT institutions, and the percentage 
of students successfully completing a college-level math course increased 25 percent on 
average (ranging from 7 to 63 percent) (Twigg, 2011). In 2012, TBR voted in favor of 
increasing the adoption of “substantially technology-driven instruction” to all public 
institutions. Colleges had until the fall of 2013 to adopt technology-driven instruction into 
their developmental math, reading, and writing courses. 


Such use of learning technology could potentially have both positive and negative 
effects on college students. Technology can be used to expand and strengthen the delivery 
of developmental math instruction (Epper & Baker, 2009). One benefit is the consistency 
it provides across sections and institutions in the delivery of math content, the means of 
practice and application, and the assessment of math knowledge. This consistency across 
students could potentially result in more even levels of academic preparation for future 
college-level courses. Another benefit concerns the use of faculty time. Because the 
hybrid emporium model is computer-based, faculty are freed up to give more 
individualized instruction to students, as little time is spent directing the class as a whole 
(Herried & Schiller, 2013). Students are also able to receive more on-demand assistance 
by individually asking questions of the faculty during class or messaging faculty through 
the online interface (Trenholm, 2006). There are also clear expectations for progress, 
including deadlines for completing exams designed to measure mastery of a given topic. 
Opportunities for practice are also a fundamental component of the model; these can help 
faculty to identify problematic areas for students and to address these concerns with them 
one-on-one. Students are able to work at their own pace, allowing some to move more 
quickly into their college-level courses. Finally, students may feel more connected or 
engaged with faculty, perceiving them as more approachable due to the more informal 
computer lab setting (Boatman & Kramer, 2019). 


Alternatively, not all students are comfortable using learning technology as an 
instructional tool, particularly in a self-driven manner. The model assumes that students 
have the ability to regulate their timing throughout the semester in order to achieve course 
milestones. It also assumes students will not be bored and become distracted by other 
websites or freedoms that come with self-driven, computer-based work. For those who 
have struggled to learn math content in the past, this independent model of learning may 
prove challenging. Students may also choose to “game” the quizzes and problem sets, 
repeatedly answering questions until they eventually land on the correct response, thus 
raising concerns about the actual level of mastery and engagement with the course 
content. Some students may also find it more difficult to establish rapport and common 
ground with instructors in the absence of direct instruction. 


In this study, I explore the relationship between technology-based developmental 
math instruction and student outcomes. I examine the academic success outcomes for 
college students who were assigned to developmental math under the hybrid emporium 
model compared to students who were assigned to developmental math under the 
traditional, lecture-based model. Using site-specific variation in the implementation of 
the hybrid emporium model across Tennessee colleges, I ask: 


1. Do technology-based developmental math courses result in higher 
course pass rates and persistence rates for students than the traditional 
version of these courses? 


2. Do these results differ between two-year (community) colleges and 
four-year colleges and by prior level of academic preparation, age, and 
gender? 


The results from this study are important to better understand what works in 
developmental education and for whom, particularly as states and systems look to 
incorporate technology in the student experience in new ways. As colleges turn to 
technology to help reduce instructional costs, the question remains whether technology 
improves student outcomes, particularly for students in developmental education. 
Technology-centered instruction remains a largely unexamined topic in the higher 
education literature, with much of the research base on computer-based instruction 
focused on middle and high schools. Given the desire for improved developmental 
education results nationwide, other states and institutions will benefit from research on 
technology-based developmental instruction in a state where the adoption of the hybrid 
emporium model has been widespread. 


2. Prior Literature 


Classroom interventions involving instructional technology can be conceptualized 
along a continuum based on the degree of reliance on technology. On one end are lighter 
technological interventions, in which instructional software is used as a supplemental tool 
in otherwise traditional courses, as opposed to using software as a primary instructional 
tool. On the other end are asynchronous online courses in which a software platform 
completely replaces all instruction. Students in these online courses have the freedom to 
complete the work when and where they wish. In between these two extremes are hybrid 
emporium courses—computer-based courses in which the instructional software delivers 
individualized instruction to students in a classroom, with an instructor in the room. 
Students are allowed to continue the work outside of class but are required to meet 
regularly together in a computer lab setting. 


While the use of technology in traditional college classrooms is not new, the 
research base on the effectiveness of its use as an instructional method remains limited. 
High school is one sector where the use of technology in classrooms has been both 
implemented and thoroughly studied. Reviewing research from both the secondary and 
postsecondary literature reveals mixed effects of the use of technology in the classroom 
on student outcomes, depending on the degree of reliance on technology. Below I 
summarize the findings from research in both the secondary and postsecondary sectors, 
organized by the degree to which technology is used for instruction. 


Research on Technology as a Supplemental Tool to Traditional 
Instruction 


Research examining the effectiveness of instructional software as a supplemental 
tool to traditional instruction finds generally positive impacts on middle and high school 
student outcomes. A review of these interventions in middle and high schools summarizes 
generally positive findings in terms of students passing the assigned math course (Tamin, 
Bernard, Borokhovski, Abrami, & Schmid, 2011; Waxman, Lin, & Michko, 2003). In a 
randomized controlled trial involving eight middle school teachers, students in math 
courses taught with Cognitive Tutor supplemental software scored higher on a common 
end-of-course algebra test and also received higher course grades than students in courses 
taught without the supplemental software (Morgan & Ritter, 2002). 


Mobile learning is one example of how technology can be used to supplement in- 
class learning that is gaining popularity in the college context. (Chen & deNoyelles, 
2013). In a randomized study of adult learners in Los Angeles, those who used a Spanish 


literacy mobile app to supplement their course content reported reading scores that were 
significantly higher than those of the comparison group (Aker, Ksoll, Miller, Perez, & 
Smalley, 2015). A similar study was conducted using mobile apps for developmental 
math students at a community college in Louisiana. Students who received access to the 
apps reported higher course grades in their developmental math courses compared to 
students in the control group, who did not get access to the apps. Treatment students 
received grades that were, on average, 0.15 grade points higher in pre-algebra and 0.23 
points higher in algebra (Giani & Martin, 2019). Results from these studies suggest that 
technology, when used as a supplemental tool to traditional instruction, can have positive 
impacts on students’ academic success. 


Research on Technology-Centered Instruction in Hybrid Courses 


In hybrid emporium courses, technology functions as more than a supplemental 
tool, replacing traditional lectures with individualized software alongside an in-class 
teacher. Across the secondary and postsecondary sectors, several studies have 
investigated the effects of “hybrid” or “blended learning,” usually defined in much the 
same way as the “hybrid emporium” model, but which may include time (typically less 
than 50 percent) spent in a traditional classroom. Only one study, conducted across three 
large urban school districts, found positive results for hybrid learning, with students 
randomly assigned to computer-based instruction scoring significantly higher on pre- 
algebra and algebra tests than students randomly assigned to traditional instruction, with 
larger test score gains for students far behind their peers academically (Barrow, 
Markman, & Rouse, 2009). The authors found no differences in the effect of hybrid 
learning by sex or race/ethnicity. 


The majority of studies on hybrid or blended learning have found null to negative 
effects on student academic outcomes. In a 2003 study, the U.S. Department of Education 
contracted with Mathematica Policy Research, Inc., and SRI International to conduct an 
experimental study of the effects of technology-based instruction in high school algebra. 
The study found that test scores in treatment classrooms (with the math technology) did 
not differ from test scores in control classrooms by statistically significant margins. These 
effects were uncorrelated with classroom and school characteristics (Dynarski et al., 
2007). Among two large-scale field experiments using computer-based instruction in high 
school, one found positive effects on student test scores in math courses (Pane, Griffin, 
McCaffrey, & Karam, 2014), while the other found null effects (Cavalluzzo, Lowther, 
Mokher, & Fan, 2012), with no evidence of positive or negative impacts for student 
subgroups defined by gender, enrollment cohort, or rurality of the school setting 
(Cavalluzzo et al., 2012). A randomized controlled field trial of high school geometry 


courses taught using technology-based instruction versus traditional instruction found 
strong negative effects on test scores for students in the hybrid courses (Pane, McCaffrey, 
Slaughter, Steele, & Ikemoto, 2010). The study also found no statistically significant 
impact on students’ attitudes toward mathematics or technology. In the community 
college context, Ryan, Kaufman, Greenhouse, She, and Shi (2016) concluded that 
students enrolled in hybrid courses perform similarly to students in a traditional 
instructional setting. 


In a descriptive study comparing the learning outcomes for introductory 
psychology students in hybrid and traditional sections at a four-year university, Powers, 
Brooks, Galazyn, and Donnelly (2016) found that exam and homework grades were lower 
in hybrid sections over the semester than in traditional sections. Similarly, in a study of 
hybrid sections of intermediate algebra at a private four-year college, Spradlin and 
Ackerman (2010) found no statistically significant differences in the posttest scores of 
students receiving traditional instruction and those receiving computer-assisted instruction. 


Two studies used a quasi-experimental analytic design to control for issues of 
student selection into courses and concluded that, compared to traditional courses, hybrid 
courses had a negative effect on the retention of information among first-year students in 
statistics courses at a four-year university (Kwak, Menezes, & Sherwood, 2015) and 
among university students enrolled in an introductory economics course (Cosgrove & 
Olitsky, 2015). In their study of students randomly assigned to either a hybrid learning or 
traditional research methods and statistics course, Goode et al. (2018) found that students 
who took the hybrid version of the course scored significantly lower on measures of 
quantitative mastery of statistical concepts than those who took the traditional version; 
however, the effect size was small. Many of the existing studies have reported small 
sample sizes and/or no equivalent comparison groups or were sponsored by an 
organization with an interest in the outcome (Dynarksi et al., 2007). 


While these studies add to the literature on the effects of hybrid courses, they do 
not specifically examine the efficacy of an instructional model that relies nearly 100 
percent on computer-based instruction, nor do they examine the context of students in 
need of developmental education. Of the studies that do focus on this population, some 
were authored with the relevant software provider, making the results subject to bias 
concerns (Perez & Foshay, 2002). Twigg (2013) descriptively reported on the outcomes 
of implementation at 32 institutions that redesigned 86 developmental math courses using 
common final exam scores, common exam items, and gains on pre- and posttests in the 
traditional and hybrid emporium formats of the courses to compare how much students 
learned in the two formats. She concluded that 71 of the redesigned hybrid emporium 
courses (83 percent) showed significant improvements over the traditional format. 


Vallade (2013) explored students’ success in intermediate and college algebra after 
receiving developmental math instruction under the hybrid emporium model at two 
Tennessee community colleges and found that these students had higher passing rates and 
mean grades and lower failure and withdrawal rates than students who took traditional 
developmental math courses. And in their descriptive study comparing outcomes from 
online, blended, and traditional developmental math courses, Ashby, Sadera, and McNary 
(2011) concluded that there were significant differences between learning environments, 
with the students in the blended courses having the least success. None of these prior 
studies, however, controlled for issues of selection bias regarding the developmental math 
students who enroll in technology-based courses. 


Two recent studies have attempted to address the issue of selection bias. An 
experimental study of a modularized, computer-assisted instructional format offered to 
developmental math students at one community college found no evidence that the 
technology-based course led to improvements in students’ enrollment, progress, or 
completion of developmental math compared to the traditional math course (Weiss & 
Headlam, 2019). The computerized developmental math course helped students make 
progress toward completing the course in the first semester, but ultimately, they were 
slightly less likely to complete the full developmental math sequence of courses. A recent 
quasi-experimental study of the effects of the hybrid emporium model in Kentucky 
community colleges reported similarly discouraging findings. Exploiting differences in 
the adoption of technology-based instruction in certain courses over time, Kozakowski 
(2019) found that using the hybrid emporium model compared to traditional instruction 
in developmental math courses reduced course pass rates, retention, and degree 
attainment in the Kentucky community college system. The effects were similar for 
different levels of developmental courses. With the exception of rigorous studies such as 
these, there is limited evidence on the impacts of the hybrid emporium model on students’ 
academic outcomes. 


Research on Online, Technology-Only Instruction 


The most intensive use of technology as a teaching tool is found in online courses. 
There is now more rigorous research on the effects of online learning in higher education 
than ever (for a recent summary, see Xu & Xu, 2019). Studies on the impacts of online 
learning on math students generally report null or negative findings on performance in 
developmental and subsequent college-level math courses. In their study of online 
summer math courses at three universities, Chingos, Griffiths, and Mulhern (2017) found 
that students in the online courses were no more likely than students in traditional fall 
math courses to enroll in higher-level courses, earn more math credits, or improve their 


grades in other math courses during the first year of college. In their study of Virginia’s 
community colleges, Xu and Jaggars (2011) reported a negative impact of enrolling in 
online introductory college-level math courses on grades in those courses. Additional 
work concluded that academically high-need students struggle in purely online courses 
(Xu & Jaggars, 2014). 


In the high school setting, many online courses exist through credit recovery 
courses, where students who need to make up credits are allowed to take a course entirely 
online outside of the school day. Recent causal research at the high school level compared 
the academic outcomes for students taking online versus face-to-face credit recovery 
courses. Researchers found no statistically significant differences in the number of math 
credits earned in four years of high school and in on-time high school graduation rates 
between students in the online and face-to-face credit recovery courses (Rickles, Heppen, 
Allensworth, Sorensen, & Walters, 2018). Another study found that among high school 
students taking a course for the first time, virtual course-taking was associated with a 
lower likelihood of taking and passing a subsequent course in the same subject (Hart, 
Berger, Jacob, Loeb, & Hill, 2019). Similarly, Heinrich, Darling-Aduana, Good, and 
Cheng (2019) found mostly negative associations between online course-taking and 
districtwide standardized math scores, with students with the lowest academic 
performance prior to the course performing the worst. And in a randomized controlled 
trial of 1,224 ninth graders who failed algebra in Chicago public high schools, students 
in an online version of the same course reported that the course was more difficult and 
received lower scores on an algebra posttest compared to those in a face-to-face section 
of the repeated course (Heppen et al., 2017). This study found no statistically significant 
differences on the likelihood of passing subsequent math courses. 


The prior literature on technology in high school and college math courses broadly 
suggests that the more technology is used to fully replace instruction, the more negative 
the impact on student grades in initial and subsequent math courses. For hybrid emporium 
courses, which rely on technology as the primary but not sole instructional tool, the results 
from prior research suggest null to negative impacts on student progress in developmental 
and college-level math courses. In the current study, I examine the effects of hybrid 
emporium courses in both community and four-year colleges in Tennessee. In addition to 
examining differences by sector, I also test whether the results differ by students’ gender, 
age, and ACT score. 


3. Policy Background and Descriptive Information 


The percentage of students taking developmental math has declined across 
Tennessee community colleges over time, from 77 percent of community college students 
enrolled in a developmental course in 2011 to 65 percent in 2018 (Tennessee Higher 
Education Commission, 2019). This decline tracks with academic policies implemented 
in the state during this time. Beginning in the fall of 2012, Tennessee eliminated non- 
credit developmental education from four-year colleges, requiring that all non-college- 
credit-bearing courses be offered only at community colleges.” Four-year colleges were 
no longer allowed to offer non-credit-bearing, stand-alone courses, and as a result these 
institutions were required to reform their developmental courses to be deemed credit- 
bearing. The six four-year colleges continued to enroll students with academic 
weaknesses, but began offering corequisite courses in which students enrolled in a three- 
credit college-level math course alongside a one- to three-credit supplemental 
developmental math section. Thus, students were still enrolled in developmental math at 
four-year colleges, only now they were simultaneously enrolled in college-level math. 
Following this change in 2012, the state legislature required the adoption of corequisite 
remediation across community colleges in 2015-16. Similar to the four-year colleges, 
students completing corequisite courses received credit for both their college-level math 
course and its accompanying developmental math section. 


In 2013-14, the state also implemented a developmental math high school 
transition course known as SAILS (Seamless Alignment in Learning Support), designed 
to reduce the number of students entering college underprepared for college-level math. 
The SAILS program scaled up over the following three years and is responsible for some 
of the decline in the number of recent high school graduates enrolling in developmental 
math in college (Kane et al., 2019). While not related to developmental education directly, 
in 2015—16, the first cohort of Tennessee Promise students graduated from high school 
and begin attending community colleges in the state tuition-free. All of these large-scale 
policy changes occurred primarily from 2012 onward and inform the developmental 
education context in Tennessee today. 


As stated earlier, in 2012 TBR also voted in favor of increasing the adoption of 


“substantially technology-driven instruction” at all public institutions. Colleges had until 


? Also in 2012, Tennessee colleges redesigned their developmental education competencies, providing 
more structure and specific content knowledge benchmarks for what students need to know before moving 
into college-level courses. 
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2013 to adopt a hybrid emporium model into their developmental math courses,* although 
several colleges had already done so. In a survey I sent to faculty and administrators 
across the 13 community colleges and six four-year colleges in 2016, respondents were 
asked to report on their timeline of adopting the hybrid emporium model. Table 1 was 
compiled from information available from TBR, The College Board, the Integrated 
Postsecondary Education Data System (IPEDS), and responses to the survey from each 
of the 19 TBR colleges. It is divided into two panels: community colleges and four-year 
colleges. This table compares the basic student population demographics of the colleges 
in the state, including the total number of students enrolled, percentage of non-White 
students, and percentage of students requiring math remediation. While there is diversity 
across the campuses on these dimensions, the difference-in-differences method of 
analysis described below is concerned with whether the institutions that adopted the 
hybrid emporium model in the earliest years (fall 2008 or earlier) are comparable to the 
institutions that adopted the model later. The institutions in Table | are used to construct 
the most observationally similar control groups of institutions for the “early adopter” and 
“late adopter” community and four-year colleges. Five community colleges and two four- 
year colleges reported adopting the hybrid emporium model at scale during or prior to 
fall 2008; these are early adopter colleges. The remaining institutions all adopted the 
model in spring 2011 or later; these are late adopter colleges. 


The content of the hybrid emporium math courses is similar to the prior traditional 
developmental math courses. The hybrid emporium courses cover material in five math 
competency areas: (1) real number sense and operations, (2) operations with algebraic 
expressions, (3) analyzing graphs, (4) solving equations, and (5) modeling and critical 
thinking. While the traditional courses covered most of these same areas, they were not 
as explicitly defined or grouped as they were under the modules used in hybrid emporium 
courses. 


In addition to surveying developmental math faculty and administrators at each 
of the 19 Tennessee public colleges, I also conducted follow-up phone interviews to 
gauge the extent to which campuses had fully implemented the model and to understand 
what instruction looked like in each of the colleges prior to the adoption of the hybrid 
emporium model. Faculty and administrators revealed that there was variation in the 
adoption of the model. For some, the adoption involved scaling up the model in different 
courses and sections over several semesters (thus leaving some courses and sections as 


3 The hybrid emporium model was thus used in developmental math when it was offered as a prerequisite 
stand-alone course and when it was offered as a corequisite section accompanying a college-level math 
course. While the full dataset used in this study includes both prerequisite and corequisite developmental 
courses, the cohorts used in the analysis (2006-07 to 2009-10) ensure that only prerequisite 
developmental math courses are used in the “control” condition and in the pre-2009 treatment condition. 
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the status quo for a period of time), while for others it involved the continued 
incorporation of some whole-class instruction into some sections permanently. 
Fortunately for the design of this study, the early adopter colleges reported a more 
consistent adoption of the hybrid emporium model, likely related to the leadership at these 
institutions and their desire to implement the model at scale. Further, only two late adopter 
colleges reported using technology in the classroom in other significant ways prior to 
their adoption of the hybrid emporium model. As a sensitivity check, I later drop these 
institutions from the analysis to check that they are not impacting the results. 


The difference between the percentage of students enrolled in traditional 
developmental courses at early versus late adopter institutions is shown in Figures | and 
2. These figures present the percentage of students enrolled in developmental math 
courses that use traditional instructional methods, averaged across the early and late 
adopter institutions, from summer 2006 to summer 2010. If all developmental math 
courses were taught using traditional instruction, both lines would be at 100 percent. 
Conversely, if all students were enrolled in hybrid emporium courses, both lines would 
be at zero percent. By definition, up until 2011, there were few sections of developmental 
math being taught using the hybrid emporium model at the late adopter colleges. 
However, at the early adopter colleges, an increasing number of developmental math 
students over this time period enrolled in hybrid emporium courses as the courses were 
scaled within their institutions. 
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Table 1 
Descriptive Information of Hybrid Emporium Model Scale-Up for the 
19 Public Colleges of the Tennessee Board of Regents, Fall 2008 


Term 
0 0 
Uber Undergraduate ery Pass eae donee 
Institution Rural, or eh % White % Black . % Full-time Graduation 4 8 Hybrid 
Enrollment Ethnic Math ; 
Suburban tees Rate (%) ee Emporium 
Minority Remediation 
Model 
Community colleges 
suburban 8,743 83 8 8 66 12 41-50 fall 2007 
2 suburban 3,335 83 7 5 68 15 61-70 spring 2008 
3 suburban 4,381 75 18 4 63 12 71-80 fall 2008 
4 urban 8,485 74 16 5 65 7 51-60 fall 2008 
5 rural 5,534 89 3 5 62 18 51-60 fall 2008 
6 urban 7,716 58 30 9 58 6 71-80 spring 2011 
7 suburban 5,470 86 3 3 67 15 41-50 fall 2011 
8 rural 4,770 83 7 7 62 17 51-60 fall 2011 
9 suburban 7,241 81 9 6 62 13 61-70 fall 2011 
10 rural 4,394 81 9 8 61 15 61-70 fall 2011 
11 rural 2,749 75 21 3 62 8 61-70 fall 2012 
12 suburban 5,918 93 2, 5 67 19 41-50 fall 2012 
13 urban 11,427 29 61 8 62 5 71-80 spring 2012 
Four-year state universities 
1 urban 8,573 66 19 10 80 36 31-40 fall 2008 
2 urban 15,823 53 36 9 78 40 31-40 spring 2008 
3 urban 6,431 23 68 3) 78 35 31-40 fall 2011 
4 rural 8,438 86 4 qs 83 52 11-20 fall 2012? 
5 rural 11,028 84 6 6 84 41 21-30 fall 2012° 
6 urban 21,252 69 18 9 81 45 31-40 Did not adopt 


NOTES: Data collected from the Tennessee Board of Regents, The College Board, IPEDS, and surveys sent to institutional administrators. Italics in the Term 
Adopted column indicate “Early adopter” colleges, which adopted the hybrid emporium model prior to 2009. All colleges used Pearson software except for 
college number 4 under the four-year colleges, which reported using Hawkes. 


“This college adopted the hybrid emporium model in its corequisite sections of developmental math, as stand-alone prerequisite courses were eliminated in 2012. 
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Figure 1 


Share of Developmental Math Students Taught Using Traditional Methods at 
Community Colleges, by Period of Hybrid Emporium Adoption 
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NOTES: Among the 13 community colleges, five institutions adopted the hybrid emporium model prior 
to 2011 (the “early adopters”), and eight adopted in 2011 or later (the “late adopters”). Descriptive 
information on the early and late adopter colleges is found in Table 1. 
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Figure 2 


Share of Developmental Math Students Taught Using Traditional Methods at 
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Four-Year Colleges, by Period of Hybrid Emporium Adoption 
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NOTES: Among the six four-year public colleges, two institutions adopted the hybrid emporium model 


prior to 2011 


(the “early adopters’), three in 2011 or later (the “late adopters”), and one did not adopt. 


15 


4. Empirical Strategy 


Data 


This study uses survey data collected from the 19 two- and four-year TBR 
colleges, transcript data from the colleges and the TBR central office, and follow-up 
phone calls, short interviews, and site visits to confirm the survey responses. Information 
about the time frame of the hybrid emporium model scale-up and the basic details of the 
reform were collected from a survey sent to the developmental education faculty and 
administrators at each of the 19 TBR institutions. Transcript data from all 19 institutions 
from school years 2006—07 to 2015—16 were provided by TBR. The data include specific 
information on student course enrollment and performance and were available for the 
periods both before and after the adoption of the hybrid emporium model. These data are 
used to evaluate student outcomes resulting from the adoption of this new model. 


Methods 


Research on the causal effects of participating in technology-based courses on 
subsequent educational outcomes has been notably absent. Much of the existing research 
compares the pass rates of students before and after the course redesigns were 
implemented, while failing to account for the selection of students into these courses and 
for any unobserved differences between them and their peers who did not take remedial 
courses. To address this issue, I adopt a difference-in-differences (DID) analytic strategy 
that involves exploiting variation in institutions’ timelines for adopting the hybrid 
emporium model to obtain an estimate of the effect of being assigned to a hybrid 
emporium developmental math course on student outcomes. 


A DID strategy has been used in other research estimating the effects of hybrid 
learning on course outcomes (Cosgrove & Olitsky, 2015; Deschacht & Goeman, 2015; 
Kwak, Menezes, & Sherwood, 2015). DID analyses do not require that researchers have 
access to all potential confounding variables. Rather, differences between the treatment 
and control groups are assumed to be fixed over time; hence, differences observed post- 
intervention can be attributed to the treatment (Zhou, Taber, Arcona, & Li, 2016). 


I first compare the outcomes for all students assigned to a developmental math 
course at an early adopter institution before and after the adoption of the hybrid emporium 
model. This difference describes the general trend in student outcomes over the past 10 
years. However, because student outcomes may have changed over time for reasons other 
than the adoption of the hybrid emporium model, I also utilize a second difference by 
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including the outcomes for students from the late adopter two- and four-year institutions 
in Tennessee that were not using this model at the time. Subtracting these two differences 
provides the estimated effect of being assigned to a hybrid emporium math course on 
student outcomes. I use the following reduced form equation: 


ay TPB [os * EMPORIUM, | + B,(POST, )+ B,(EMPORIUM, ) 


+ B,(Z,,)+ Sing +E 


where Y; refers to one of the outcomes at time ¢ for institution 7. These outcomes include 
student performance on early college outcome variables, including the number of credit 
hours accumulated after the first and second year, the number of terms to completion of 
developmental math (a measure of progression through developmental math), persistence 
to the second and third year of college, passing a college-level math course,* and degree 
or certificate completion. I estimate the impact on completing an associate degree (for 
students attending two-year colleges); a bachelor’s degree (for students attending four- 
year colleges); and any credential, including a certificate, an associate degree, or a 
bachelor’s degree combined (for students attending both two-year and four-year 
colleges); all within six years. POST; is a dichotomous variable equal to one if a student 
is assigned to a developmental math course for the first time in the years after the adoption 
of the hybrid emporium model at her institution and zero otherwise. EMPORIUM; is 
equal to one for students enrolled in an early adopter institution and zero for students 
attending any other two- or four-year public TBR institution in Tennessee. Z, is a vector 
of student characteristics including gender, race/ethnicity, and prior academic 
performance (as measured by the junior-year ACT math and English scores and high 
school GPA). The DID model would be biased by unobserved college-level factors that 
differ between early and late adopter schools and influence student performance on any 
of the outcome variables. To address this concern, I add to the model an institution-level 
fixed effect to compare students’ prior outcomes to average student outcomes in late 
adopter colleges. When early and late adopter colleges have a similar pretreatment trend, 
the DID model represents the counterfactual change of adopting a hybrid emporium 
model. Therefore, ~’ represents individual dummy variables for each institution to 
capture institution-level fixed effects, and ey represents the residual. 


Equation (1) is akin to the reduced-form model for an instrumental variables 
approach in which being assigned to a developmental math course is used as an 
instrument for whether a student ever enrolled in a developmental math course and 


4 Over 90 percent of students took one of three college-level math courses: Contemporary Math 
(beginning algebra), College Algebra, or Introductory Statistics. 
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whether the student attended a college that offered the hybrid emporium version during 
that year, using college and year fixed effects to account for temporal trends and 
institutional-level variation. This provides the policy-relevant intent-to-treat estimates of 
the effect of the hybrid emporium model on student outcomes. As is common for DID 
analyses, I also test for common trends and make other robustness checks, as described 
below. 


Several issues and assumptions are relevant in DID studies. Serial correlation is 
one such issue (Bertrand, Duflo, & Mullainathan, 2004). When multiple time points of 
data are collected for the same person or unit of analysis, the error terms will be 
correlated. This leads the estimate of the standard error to be underestimated, leading to 
a higher Type I error rate. To adjust for serial correlation, I utilize a block bootstrap 
method to account for clustering at the time points for each person. A second common 
issue in DID studies is related to the stable unit treatment value assumption, which holds 
that whether Person A receives the treatment should not affect the outcome for Person B 
(Lechner, 2011). A third issue in DID analysis relates to the exogeneity assumption, 
which is violated when the covariates in the analysis are also affected by the treatment. 
In the context of this study, this could occur if students somehow knew they would be 
required to take a hybrid emporium developmental math course, which then changed their 
behavior such that measures of the control variables are actually endogenous. Finally, the 
common trend assumption in DID studies holds that the trend over time for the outcome 
variable of interest would be the same for the treatment and control groups if the 
intervention had not occurred. In Section 5, I describe the robustness checks I conducted 
to address these concerns. 


In addition to the overall treatment effect, I examine several heterogeneous 
treatment effects, including gender, age, and incoming level of math preparedness, as 
measured by the ACT math exam. The hybrid emporium model may affect men and 
women differently for several reasons. Historically, more women enroll in developmental 
math than men, particularly at the lowest levels of developmental math (Hagedorn, 
Siadat, Fogel, Nora, & Pascarella, 1999). Prior research suggests that there is an important 
relationship between gender and math self-concept and math performance (Cadinu, 
Maass, Rosabianca, & Kiesner, 2005; Sax, 1994), which begins developing as early as 
elementary school (Cvencek, Meltzoff, & Greenwald, 2011). Among college students, 
Sax (1994) found statistically significant differences between the math self-concepts of 
male and female students, with women reporting lower self-concepts than men 
throughout college. Taking more math and science courses has a positive effect on both 
male and female students’ math self-concepts. Math self-concept has been shown to be a 
positive predictor of math course grades and majoring in STEM (Sax, Kanny, Riggers- 
Piehl, Whang, & Paulson, 2015), and women are less likely than men to complete a 
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STEM major within six years of enrolling in college, even though women report higher 
grade point averages (Gayles & Ampaw, 2014). Female college students who do not need 
developmental math are more likely to graduate with a STEM major than those who do 
need to take developmental math courses (Gayles & Ampaw, 2014). Additionally, fifteen 
years ago it was generally the case that women had less computer experience than men 
(Margolis & Fisher, 2003). 


Because the hybrid emporium model relies on technology-based instruction, there 
may be differential impacts of the intervention across age groups. Nationally, 62 percent 
of students 24 or older at public two-year institutions and 66 percent of students 24 or 
older at public four-year institutions enroll in a remedial course. For students 18 or 
younger, the numbers are 69 percent and 37 percent, respectively (U.S. Department of 
Education, n.d.). Students 24 or older are more likely to persist in online courses (Park & 
Choi, 2009), despite similar preparedness and attitudes toward computers across age 
groups (Broady, Chan, & Caputi, 2010; Wagner, Hassanein, & Head, 2010). Older 
students also perform better in online courses compared to their expected outcomes in 
comparable face-to-face courses, suggesting that younger students may need more 
support in a technology-based instructional environment (Wladis, Conway, & Hachey, 
2015). 


Finally, the hybrid emporium model may impact students with higher levels of 
academic need differently than students who are stronger academically. Prior research 
shows that interventions can result in differential outcomes when grouping students by 
their incoming test scores (Boatman & Long, 2018; Dadger, 2012; Hodara, 2015; Xu, 2016). 


Sample 


The analytic sample includes cohorts from all TBR colleges from 2006-07 to 
2009-10, prior to the adoption of the hybrid emporium model at the late adopter 
institutions. The outcomes for all cohorts are then tracked until 2015-16, offering six 
years of follow-up data. 


Tables 2 and 3 describe the characteristics of students who began at early versus 
late adopter colleges, for community colleges and four-year colleges, respectively. 
Overall, the early and late adopter institutions look similar to one another, though there 
are a few small differences, such as average ACT scores and the share of students scoring 
below 19 on ACT math. These differences suggest the late adopter colleges may serve 
slightly lower achieving students, although these differences are substantively small. 
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Table 2 


Community College Student Characteristics by 
Timing of Hybrid Emporium Adoption, Fall 2008 


Ever Enrolled ina 


All Students Developmental Math Course 


ACT Math Score < 19 


Early Late 
Adopter Adopter Early Adopter Late Adopter Early Adopter Late Adopter 
Bae ae Institutions Institutions Institutions Institutions 
Institutions Institutions 
(1) (2) (3) (4) (5) (6) 
Female 0.550 0.597 0.593 0.619 0.613 0.666 
White 0.707 0.674 0.633 0.618 0.631 0.623 
Black 0.162 0.203 0.227 0.266 0.236 0.248 
Age at 23.38 23.99 18.95 19.19 23.57 23.62 
college entry (7.99) (8.41) (1.44) (2.22) (7.98) (6.99) 
19.38 18.62 16.11 15.87 16.07 15.94 

ee mann (4.07) (4.00) (1.42) (1.49) (1.65) (1.68) 

[5,323] [11,452] --- --- [1,395] [3,559] 
HS 2.98 2.93 2.76 2.71 2.83 2.80 
GPA/GED (0.740) (0.836) (0.638) (0.676) (0.832) (0.819) 

intile of 

ee 277 2.64 2.64 2.58 2.63 2.43 
odaancatie (1.35) (1.45) (1.35) (1.46) (1.33) (1.40) 
(1=lowest) 
Number ot 7,935 17,859 2,681 6,730 2,007 5,191 
students 
Number of 5 g 5 g 5 g 
colleges 


NOTES: The table displays means, with standard deviations in parentheses. The analytic sample includes data from all 
community colleges. ACT data is missing for some of the students in the sample. The number of observations for those 
with ACT math scores is in brackets. 
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Table 3 


Four-Year College Student Characteristics by Timing of Hybrid Emporium Adoption, Fall 2008 


Ever Enrolled in a Developmental 


All Students ACT Math Score < 19 Matin Course 
Early Adopter Late Adopter Early Adopter Late Adopter Early Adopter Late Adopter 
Institutions Institutions Institutions Institutions Institutions Institutions 
(1) (2) (3) (4) (5) (6) 

Female 0.598 0.546 0.710 0.610 0.732 0.603 
White 0.571 0.605 0.519 0.501 0.514 0.478 
Black 0.292 0.283 0.204 0.236 0.220 0.183 
Age at 22.60 21.82 18.96 18.88 20.70 20.40 
college entry (7.03) (7.51) (1.27) (1.76) (3.23) (4.29) 

20.43 20.20 16.47 16.19 16.17 16.20 
ACT math 
score (4.15) (4.48) (1.19) (1.33) (1.24) (1.50) 

[3,453] [3,730] --- --- [549] [917] 
HS GPA/ 3.15 3.14 3.03 2.90 2.99 2.90 
GED (0.60) (0.58) (0.48) (0.50) (0.48) (0.51) 
Quintile of 
median ZIP 3.39 2.56 3.00 2.50 2.82 2.43 
code income (1.44) (1.40) (1.51) (1.40) (1.66) (1.37) 
(1=lowest) 
Numbenoh 4,835 4,965 1,381 1,648 597 1,060 
students 
Number of > 3 > 3 2 3 
colleges 


NOTES: The table displays means, with standard deviations in parentheses. The analytic sample includes data from the 
five TBR four-year universities which eventually adopted the hybrid emporium model. ACT data is missing for some of 
the students in the sample. The number of observations for those with ACT math scores is in brackets. 
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5. Findings and Discussion 


Results 


The main results for both the community colleges (column 1) and four-year 
colleges (column 3) are presented in Table 4, along with the comparison means for 
students attending late adopter institutions. Among community college students, hybrid 
emporium model developmental math courses did not result in statistically different pass 
rates for their developmental math course or statistically significant differences in the 
number of terms to completion of the developmental math course when compared to the 
traditional developmental course. However, under hybrid emporium courses students 
were 5.7 percentage points less likely to pass their first college-level math course; they 
also completed 1.6 fewer credits over six terms (from a comparison mean of 23.5 credits). 
Additionally, hybrid emporium students at community colleges were 3.5 percentage 
points less likely to complete an associate degree and 3.7 percentage points less likely to 
complete any credential within six years. When compared to a base credential completion 
rate of 20.8 percent, this is a sizable effect. 


Among students beginning in four-year colleges, being assigned to a 
developmental math course taught using technology-based instruction led to higher rates 
of passing the developmental math course and completing the course in 0.5 fewer terms, 
compared to students assigned to traditional developmental math courses. However, 
similar to the results from community colleges, students assigned to hybrid emporium 
courses at the four-year colleges were 5.4 percentage points less likely to pass their first 
college-level math course and 5.0 percentage points less likely to persist from the second 
to the third year of college. There were no statistically significant differences in the 
number of credits completed over time or degree attainment. These findings suggest that 
students at four-year institutions benefit from the self-paced, technology-centered 
teaching when it comes to passing developmental math, but this positive impact does not 
carry through into college-level math. 


Table 5 presents the impact estimates for instruction under the hybrid emporium 
model in community colleges by gender and age. While both female and male students 
who were assigned to hybrid emporium developmental math courses had lower pass rates 
in college-level math when compared to students assigned to the traditional 
developmental math course, these negative effects were slightly more pronounced for 
female students. However, male students assigned to hybrid emporium courses 
experienced reductions in number of credits earned over time and in earning an associate 
degree or any credential within six years, whereas there was no statistically significant 
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effect for female students assigned to hybrid emporium courses on the number of credits 
completed or degree completion. 


Columns 5-8 of Table 5 present the results for students beginning community 
college under age 23 compared to students 23 and older. Those 23 and older who were 
assigned to hybrid emporium developmental math courses spent 0.5 additional terms 
completing the developmental math course, but were no more or less likely to pass their 
first college-level math course compared to similarly aged students assigned to traditional 
developmental math courses. The effects observed at community colleges overall are 
largely driven by traditionally aged students, those under 23. 


At the four-year colleges (Table 6), male and female students generally had 
similar outcomes as students at community colleges. Interestingly, older students who 
were assigned to hybrid emporium courses had higher pass rates in developmental math. 
They also spent fewer terms in developmental math compared to students under age 23. 


Table 7 presents the results for those with ACT math scores 16 and below 
compared to those with ACT math scores above 16. At the community colleges, the 
negative student outcomes associated with the hybrid emporium model are largely 
concentrated among those with ACT math scores above 16. At the four-year colleges, 
there are fewer substantial differences by ACT score, although there are positive effects 
on credential completion among students with low ACT math scores. So, while the model 
may not be a significant improvement over traditional instruction, it is not harming 
students with the lowest math skills. 
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Table 4 


Coefficient Estimates of the Effect of a Hybrid Emporium Developmental Math 
Course on Course Pass Rates, Persistence, and Degree Attainment for 
Cohorts 2006—07 Through 2009-10 


Two-Year College Four-Year College 
Two-Year Comparison Mean Four-Year Comparison Mean 
College (at Late Adopter College (at Late Adopter 
Colleges) Colleges) 
Dependent Variable (1) (2) (3) (4) 
-0.010 0.639 0.054** 0.781 
Passed developmental math (0.028) (0.025) 
Terms to completion of -0.240 2.01 -0.502*** 1.42 
developmental math (0.212) (0.093) 
Passed first college-level math -0.057** 0.412 -0.054*** 0.425 
course (0.022) (0.015) 
Cumulative credits within 3 -0.722* 16.27 0.801 23.05 
semesters (0.379) (0.497) 
Cumulative credits within 6 -1.556*** 23.52 -1.059 40.76 
semesters (0.575) (0.940) 
; -0.065*** 0.450 0.006 0.485 
Retention from year | to year 2 (0.015) (0.018) 
: -0.003 0.265 -0.050*** 0.447 
Retention from year 2 to year 3 (0.014) (0.015) 
Earned associate degree within -0.012 0.056 0.003 0.006 
3 years (0.011) (0.002) 
Earned associate degree within -0.035** 0.132 0.001 0.018 
6 years (0.014) (0.010) 
Earned bachelor’s degree -0.008 0.047 0.056 0.433 
within 6 years (0.005) (0.032) 
Earned any credential within 6 -0.037** 0.208 0.056 0.463 
years (0.015) (0.032) 
Number of observations 18,743 5,500 


NOTES: Standard errors in parentheses. The “any credential” outcome includes certificates, associate degrees, 
and bachelor’s degrees. 
*p <.05, ** p< .01, *** p< .001. 
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Table 5 


Community College Coefficient Estimates of the Effect of a Hybrid Emporium Developmental Math Course on 
Course Pass Rates, Persistence, and Degree Attainment for Cohorts 2006—07 Through 2009-10, by Gender and Age 


Gender 
Control Control 
Women Mean Men Mean 
Dependent Variable (1) (2) (3) (4) 
-0.003 0.681 -0.019 0.565 
Passed developmental math (0.031) (0.047) 
Terms to completion of -0.135 1.97 -0.167 2.11 
developmental math (0.143) (0.174) 
Passed first college-level math -0.050* 0.624 -0.059 0.577 
course (0.028) (0.055) 
Cumulative credits within 3 -0.600 16.74 -0.833* 15.42 
semesters (0.530) (0.487) 
Cumulative credits within 6 -1.354 24.16 -1.598* 22.36 
semesters (0.940) (0.895) 
: -0.063*** 0.451 -0.060** 0.440 
Retention from year | to year 2 (0.014) (0.026) 
: 0.026 0.284 -0.052 0.215 
Retention from year 2 to year 3 (0.019) (0.035) 
Earned associate degree within 3 -0.009 0.059 -0.014 0.053 
years (0.016) (0.011) 
Earned associate degree within 6 -0.016 0.138 -0.056*** 0.122 
years (0.017) (0.018) 
Earned bachelor’s degree within 6 0.000 0.047 -0.021* 0.047 
years (0.007) (0.00) 
Earned any credential within 6 -0.010 0.211 -0.068*** 0.203 
years (0.020) (0.017) 
Number of observations 11,921 6,822 


< 23 Years 
Old 
(5) 
-0.008 
(0.028) 
-0.243 
(0.143) 
-0.055** 
(0.021) 
-0.583 
(0.381) 
-1.405** 
(0.577) 
-0.066*** 
(0.015) 
-0.001 
(0.015) 
-0.012 
(0.011) 
-0.035** 
(0.014) 
-0.008 
(0.005) 
-0.038** 
(0.015) 
11,269 


Control 
Mean 


(6) 
0.635 


1.84 
0.606 
16.21 
23.46 
0.462 
0.266 
0.056 
0.132 
0.047 


0.207 


Age 
23 Years or 
Older 
(7) 
-0.062 
(0.146) 
0.518** 
(0.201) 
0.142 
(0.106) 
-5.536 
(3.477) 
-8.702 
(6.267) 
0.310 
(0.210) 
0.348 
(0.606) 
0.134 
(0.112) 
0.088 
(0.069) 
0.018 
(0.035) 
0.094 
(0.073) 
7,474 


NOTES: Standard errors in parentheses. The “any credential” outcome includes certificates, associate degrees, and bachelor’s degrees. 


* p< .05, ** p< .01, *** p< 001. 


25 


Control 
Mean 


(8) 
0.763 


1.69 
0.692 
18.37 
25.15 
0.432 
0.279 
0.068 
0.158 
0.029 


0.229 


Table 6 


Four-Year College Coefficient Estimates of the Effect of a Hybrid Emporium Developmental Math Course on 
Course Pass Rates, Persistence, and Degree Attainment for Cohorts 2006—07 Through 2009-10, by Gender and Age 


Dependent Variable 
Passed developmental math 


Terms to completion of 
developmental math 


Passed first college math 


Cumulative credits within 3 
semesters 


Cumulative credits within 6 
semesters 


Retention from year | to year 2 


Retention from year 2 to year 3 


Earned associate degree within 3 
years 


Earned associate degree within 6 
years 


Earned bachelor’s degree within 6 
years 


Earned any credential within 6 
years 


Number of observations 


< 23 Years 
Old 
(5) 
0.050* 
(0.025) 
-0.296** 
(0.130) 
-0.053*** 
(0.014) 
0.889* 
(0.473) 
-0.863 
(0.869) 
0.011 
(0.017) 
-0.052*** 
(0.015) 
0.003 
(0.002) 
0.000 
(0.010) 
0.056* 
(0.028) 
0.056* 
(0.028) 
4,739 


Control 
Mean 


(6) 
0.781 
2.024 
0.486 
22.93 

40.613 
0.499 
0.472 
0.006 
0.018 


0.434 


0.464 


Gender 
Control Control 
Women Mean Men Mean 
(1) (2) (3) (4) 
0.042 0.824 0.074 0.709 
(0.042) (0.044) 
-0.517** 1.13 -0.219*** 1.45 
(0.239) (0.078) 
-0.056** 0.501 -0.036 0.453 
(0.021) (0.029) 
0.955* 23.87 0.091 21.66 
(0.481) (0.806) 
-1.091 42.48 -1.844 37.85 
(1.044) (2.207) 
0.012 0.541 0.000 0.413 
(0.018) (0.036) 
-0.048** 0.498 -0.061** 0.402 
(0.023) (0.025) 
0.002 0.008 0.006* 0.003 
(0.002) (0.003) 
0.004 0.022 -0.012 0.011 
(0.011) (0.013) 
0.077 0.473 -0.027 0.366 
(0.129) (0.030) 
0.085 0.506 -0.021 0.390 
(0.049) (0.034) 
3,664 1,836 


Age 
23 Years or 
Older 
(7) 
0.271** 
(0.114) 
-1.189*** 
(0.106) 
0.581 
(0.408) 
-10.17 
(8.45) 
-15.54 
(9.85) 
-0.386 
(0.360) 
0.065* 
(0.030) 
0.003 
(0.005) 
0.009 
(0.006) 
0.073 
(0.354) 
0.073 
(0.354) 
761 


NOTES: Standard errors in parentheses. The “any credential” outcome includes certificates, associate degrees, and bachelor’s degrees. 


* p< .05, ** p< .01, *** p< 001. 
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Control 
Mean 


(8) 
0.848 

1.70 
0.472 
34.59 
54.80 
0.489 
0.423 
0.001 
0.008 


0.409 


0.409 


Table 7 


Coefficient Estimates of the Effect of a Hybrid Emporium Developmental Math Course on 
Course Pass Rates, Persistence, and Degree Attainment for Cohorts 2006—07 Through 2009-10, by ACT Score 


Dependent Variable 
Passed first developmental math 


Terms to completion of 
developmental math 


Passed first college-level math 
course 


Cumulative credits within 3 
semesters 


Cumulative credits within 6 
semesters 


Retention from year | to year 2 


Retention from year 2 to year 3 


Earned associate degree within 3 
years 
Earned associate degree within 6 
years 


Earned bachelor’s degree within 6 
years 


Earned any credential within 6 
years 


Number of observations 


ACT Math 
<= 16 
(1) 
-0.018 
(0.039) 
-0.216** 
(0.079) 
-0.053 
(0.036) 
-0.640 
(0.490) 
-1.170* 
(0.679) 
-0.065*** 
(0.018) 
0.001 
(0.021) 
0.003 
(0.011) 
-0.028 
(0.018) 


-0.005 
(0.005) 


-0.024 
(0.018) 
12,990 


Two-Year Colleges 


Control 
Mean 


(2) 
0.637 


2.14 
0.401 
15.94 
22.74 
0.401 
0.247 
0.042 
0.112 


0.036 


0.181 


ACT Math 
> 16 
(3) 
0.010 
(0.017) 


-0.009 
(0.109) 


-0.064*## 
(0.020) 


-0.818* 
(0.482) 


-2.073** 
(0.920) 


-0.065**# 
(0.017) 


-0.007 
(0.024) 


-0.040*## 
(0.010) 


-0.046** 
(0.019) 


-0.011 
(0.010) 


-0.062*** 
(0.022) 


5,753 


Control 
Mean 


(4) 
0.644 


1.52 
0.461 
17.06 
25.33 
0.474 
0.261 
0.090 
0.179 


0.071 


0.271 


ACT Math 
<= 16 
(5) 
0.062** 
(0.030) 
-0.399*** 
(0.111) 
-0.050** 
(0.021) 
0.903 
(0.717) 
-1.877 
(1.650) 
-0.023 
(0.031) 
-0.046*** 
(0.014) 
0.001 
(0.002) 
-0.001 
(0.007) 


0.070* 
(0.031) 


0.072* 
(0.035) 


3,523 


Four-Year Colleges 


Control 
Mean 


(6) 
0.795 


1.76 
0.446 
22.84 
40.64 
0.421 
0.415 
0.002 
0.014 


0.420 


0.444 


ACT Math 
> 16 
(7) 
0.041 
(0.036) 
0.004 
(0.044) 
-0.081** 
(0.035) 
0.562 
(1.005) 
1.270 
(1.358) 
0.081*** 
(0.016) 
-0.060* 
(0.031) 
0.009 
(0.005) 
-0.008 
(0.021) 


0.022* 
(0.012) 


0.010 
(0.028) 
1,977 


Control 
Mean 


(8) 
0.761 


1.09 
0.521 
23.35 
40.93 
0.498 
0.466 
0.011 
0.024 


0.453 


0.491 


NOTES: Standard errors in parentheses. The “any credential” outcome includes certificates, associate degrees, and bachelor’s degrees. 


* p< .05, ** p< .01, *** p< 001. 
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Robustness Checks 


Tables 8 and 9, along with Figures 3 and 4, present the results for three common 
robustness checks for the difference-in-difference analyses. Table 8 presents the results 
from a falsification test, in which I drop data from the actual first term of early hybrid 
emporium implementation and instead randomly assign a false term from the years prior 
to adoption. For example, if a college adopted the hybrid emporium model at scale in fall 
2008, this school was randomly assigned a false adoption term of spring 2007 or fall 
2007, etc. Since this test includes artificial adoption terms and excludes data from the 
post-hybrid emporium adoption terms, I would not expect to find significant results unless 
factors unrelated to the hybrid emporium model led to changes in the outcomes. As shown 
in Table 8, I do not observe any statistically significant effects for this falsification test, 
which improves confidence in the findings from the main analyses. 


Table 9 presents a covariance balance check, in which I treat control variables as 
outcomes for the analysis. The goal of this covariate balance check is to estimate whether 
the observed effects may be driven, in part, by changes in the control variables over time. 
Examining gender, age at college entry, ACT math score, and high school GPA, I observe 
two statistically significant coefficients for gender and age at the community colleges. 
This suggests that the composition of the student body at the early adopter community 
colleges became more male and younger over time compared to the late adopter 
community colleges. This could be due to a statewide shift in the percentage of female 
students and the average age of students at all Tennessee colleges over time, with more 
female students enrolling at late adopter colleges, for example. Additionally, the 
magnitude of these differences is small relative to the mean values for late adopter 
schools. 


In Figures 3 and 4, I test for the presence of time trends in the terms prior to and 
after hybrid emporium adoption at early adopter colleges. I display graphs for six of the 
11 outcomes, although I observe similar trends for the outcomes not pictured. The vertical 
line represents the term of adoption of the hybrid emporium model at each of the 
institutions in the study. The coefficients to the left of the vertical line in Figures 3 and 4 
capture the trend, by term, for each outcome prior to the adoption of the hybrid emporium 
model across two-year and four-year colleges, respectively. Detecting significant trends 
in the terms prior to adoption of the model would suggest that significant effects observed 
in the actual term of adoption could be driving the results, or could be due to chance 
variation or prior long-running trends. The gray lines above and below the center line 
represent the 95 percent confidence interval. Fortunately, for both community and four- 
year colleges, I observe that the estimates are stable in the pre-adoption terms. I observe 
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only one statistically significant trend prior to the adoption of the hybrid emporium 
model, which is the completion of any credential within six years four terms prior to the 
adoption of the hybrid emporium model at four-year colleges. This is the only significant 
coefficient at the 5 percent level, which is reasonable to assume occurred due to chance. 


The coefficients to the right of the vertical line allow for the effects of the hybrid 
emporium model to vary by term. For the early adopter colleges it may be the case that 
the direction and/or magnitude of the outcomes changes as the hybrid emporium model 
has been adopted for more terms. Generally, at the community colleges (Figure 3) I 
observe the outcomes to be fairly stable in the post-adoption terms. At the four-year 
colleges (Figure 4) I observe a decline in the terms to completion of developmental math 
in the post-adoption terms. I also observe a sharp drop in year two enrollment at four- 
year colleges in the second term after adoption and an increase in credits accumulated, 
which then return to the prior averages in subsequent terms. Overall, the event study 
analysis captured in Figures 3 and 4 reveals no substantive concerns with pre-treatment 
trends, and little evidence of changes in the outcomes in subsequent terms post-adoption. 
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Table 8 


Falsification Test (Random Offset Term) Coefficient Estimates of Hybrid Emporium 
Model, Entrants Through Summer 2007 


Two-Year Colleges Four-Year Colleges 


Dependent Variable () @) 
-0.037 -0.159 
Passed first developmental math (0.027) (0.246) 
Terms to completion of developmental -0.060 0.041 
math (0.088) (0.090) 
-0.002 -0.024 
Passed first college math (0.046) (0.061) 
hoursa -0.613 -1.046 
Cumulative credits within 3 semesters (0.425) (1.570) 
; beets -0.065 -2.632 
Cumulative credits within 6 semesters (0.811) (1.484) 
-0.005 0.040 
Retention from year | to year 2 (0.020) (0.079) 
0.032 -0.027 
Retention from year 2 to year 3 (0.047) (0.048) 
ae 0.004 0.002 
Earned associate degree within 3 years (0.012) (0.002) 
— -0.014 0.000 
Earned associate degree within 6 years (0.021) (0.005) 
Earned bachelor’s degree within 6 0.015 0.012 
years (0.012) (0.042) 
ead -0.004 -0.003 
Earned any credential within 6 years (0.015) (0.037) 
Number of observations 3,821 ed 


NOTES: Standard errors in parentheses. 
*p <.05, ** p< .01, *** p< .001. 
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Table 9 


Reduced-Form Estimates of Hybrid Emporium Model Participation 
on Covariate Balance 


Two-Year Colleges Four-Year Colleges 


Mean Values Mean Values for 


Reduced-Form for Late Reduced-Form 
: : Late Adopter 
Estimate Adopter Estimate 
Schools 
Dependent Variable Schools 
Beraale -0.039*** 0.644 0.013 0.627 
(0.010) (0.019) 
dere iuscen te -0.072** 19.12 -0.056 18.58 
& ates (0.030) (0.059) 
-0.059 15.73 0.094* 16.13 
ACT math score (0.039) (0.051) 
0.059 2.68 0.050 2.90 
neon (0.040) (0.046) 
Number of observations 18,875 5,541 


NOTES: Race/ethnicity variables are represented as proportions. ACT math score is on a 1-36 scale. 
* p< .05, ** p< .01, *** p< .001. 
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Figure 3 


Event Study Analysis for Community Colleges, Five Terms Pre- and Post- Hybrid Emporium Model Adoption 


Pass First Dev Math Pass First College Math Terms to Complete Dev Math 
® oe 


.. 


A 


Point estimate w/ 95% confidence interval 


Term Relative to 1st Emporium term _Term Relative to 1st Emporium term —_‘Term Relative to 1st Emporium term 


Cumulative Credits, 6 Terms Enroll in Year 2 Any Credential in 6 years 


Term Reiative to 1st Emporium term Term Relative to 1st Emporium term —_Term Reiative to 1st Emporium term 


NOTES: The vertical line represents the term of adoption of the hybrid emporium model. The coefficients to the left of the vertical line capture the 
trend, by term, for each outcome prior to the adoption of the hybrid emporium model across community colleges. Two terms before the adoption of 
the hybrid emporium model is omitted in the analysis. 
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Figure 4 


Event Study Analysis for Four-Year Colleges, Five Terms Pre- and Post- Hybrid Emporium Model Adoption 


Pass First Dev Math Pass First College Math Terms to Complete Dev Math 


Point estimate w/ 95% confidence interval 


5+ +1 42 «43 «44 «445 +2 43 +4 «46 +2 +3 +4 «45 
Term Relative to ‘ist Emporium Term Term ‘Relative to 1st ‘Emporium Term Term | Relative to 1st ‘Emporium Term 
Cumulative Credits, 6 Terms Enroll in Year 2 Any Credential in 6 Years 


+1 +20 +3 +448 +4 45 +20 430 +4 «45 
Tem | Relative to 1st Emporium Term Term F Relative to 1st ‘Emporium Term Term | Relative to 1st ‘Emporium Term 


NOTES: The vertical line represents the term of adoption of the hybrid emporium model. The coefficients to the left of the vertical line capture the trend, by 
term, for each outcome prior to the adoption of the hybrid emporium model across four-year colleges. Two terms before the adoption of the hybrid emporium 
model is omitted in the analysis. 
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Discussion 


In this study, I assess the impact of using the hybrid emporium model for 
developmental math courses on students’ academic outcomes at both two-year and four- 
year public colleges in Tennessee. 


Community colleges 


Among students attending community colleges, there appear to be no significant 
differences in the pass rates for developmental math courses or in the number of terms 
spent in developmental math between students who were assigned to a hybrid emporium 
developmental math course and those assigned to a traditional developmental math 
course. However, hybrid emporium students were 5.7 percentage points less likely to pass 
their first college-level math course and earned 1.6 fewer credits over six terms (from a 
comparison mean of 23.5 credits). Under a hybrid emporium model students were 3.5 
percentage points less likely to complete an associate degree and 3.7 percentage points 
less likely to complete any credential within six years. These findings on passing college- 
level math and on associate degree attainment are similar in magnitude to what 
Kozakowski (2019) found in her study of the effects of technology-based instruction at 
Kentucky community colleges. 


Among students attending community colleges, assignment to a hybrid emporium 
developmental math course reduced the likelihood that students would pass their first 
college-level math course (this was also the case among four-year college students—see 
below). For the main analysis, all three potential first college-level math courses 
(beginning algebra, college algebra, and introductory statistics) are combined, but the 
negative results are similar when isolating the first college-level math course to one of 
these three courses at both two-year and four-year colleges, suggesting that type of first 
college-level math course is not relevant to the results. Thus this negative finding is a 
puzzle, as it is not clear whether it is related to a negative effect of the technology or to 
the fact that in college-level math courses, students who had become accustomed to 
technology-based instruction in their developmental course then returned to a traditional 
format in the college-level course. One explanation for why students assigned to 
developmental math under the traditional model may perform better in their first college- 
level math course could simply be that the instructional methods of both courses are 
aligned. Future research examining the effects of technology-centered developmental 
math instruction should consider the effect of transitioning from one instructional mode 
to another between developmental and college-level courses. 
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In examining the effects of the hybrid emportum model for various subgroups of 
students at community colleges, I find that the lower overall pass rates in college-level 
math were largely driven by female rather than male students. Male students experienced 
greater reductions than female students in the likelihood of earning an associate degree 
or any credential within six years but were not more statistically likely to fail their first 
college-level math course. There were no statistically significant effects for female 
students in a technology-based developmental math course on the number of credits 
completed over time, or on credential completion. 


I also find that the effects observed at community colleges were largely driven by 
younger rather than older students. These findings support earlier research that found that 
older students perform better in online courses compared to their expected outcomes in 
traditional courses (Wladis, Conway, & Hachey, 2015). I also observe that the negative 
student outcomes associated with the hybrid emporium model were largely concentrated 
among those with ACT math scores above 16. Community college faculty in Tennessee 
were initially concerned about the impact of this type of instructional model on students 
with very low ACT math scores, but results from this study suggest that students with 
lower ACT math scores were, broadly, not disadvantaged by exposure to the model. The 
negative findings for students close to the cutoff is consistent with other quasi- 
experimental research in developmental education that uses a regression discontinuity 
design, comparing the outcomes for students directly on either side of the placement 
cutoff (Boatman & Long 2018; Clotfelter, Ladd, Muschkin, & Vigdor 2015; Dadgar, 
2012; Hodara, 2015; Martorell & McFarlin, 2011; Melguizo, Bos, Ngo, Mills, & Prather, 
2016; Scott-Clayton & Rodriguez, 2015; Xu & Dadgar 2018). While these prior studies 
do not specifically examine the effects of the hybrid emporium model, they do align with 
these results, suggesting that developmental education students with placement test scores 
close to the cutoff experience null to negative outcomes. 


Four-Year Colleges 


With respect to four-year college students, results from this study show that under 
a hybrid emporium model students had higher rates of passing developmental math, on 
average, and spent 0.5 fewer terms in developmental math compared to those assigned to 
a traditional course. Similar to the community college students, students at four-year 
colleges were 5.4 percentage points less likely to pass their first college-level math 
course, and they were 5.0 percentage points less likely to persist from the second to the 
third year. There were no statistically significant differences in the number of credits 
completed over time or on degree attainment at the four-year colleges. Further, there were 
no substantive differences by gender or by ACT math score. Older students reported 
higher pass rates in their developmental math courses under the hybrid emporium model, 
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and also spent fewer terms in developmental math compared to students aged younger 
than 23 years of age. 


Limitations of the Study 


While this study attempts to control for important differences in students who 
were assigned to and institutions that adopted technology-centered courses over time, 
several caveats should be acknowledged. First, this study examines the adoption of the 
hybrid emporium model of teaching developmental math in the earliest years of 
implementation in Tennessee. As is common in the adoption of new programs and 
technologies, much is learned and adjusted from the challenges in the early years. 
Qualitative interviews with faculty at six of these institutions revealed a consistent degree 
of skepticism about the success of this method of instruction in the early years of 
adoption. Student success and retention is based on a number of factors, but interactions 
with faculty are a critical component (Tinto, 1993). If faculty were demonstrating their 
own concerns and growing pains regarding this new style of teaching during this time, it 
is reasonable to assume that this may have influenced student success in the early years 
of adoption. As faculty and administrators become more facile with technology serving 
as the primary instructional tool, the impacts on student success may improve. With the 
difference-in-differences design, the impacts of the instructional method four or five 
years into implementation are, by definition, unknown. 


This study also treats the hybrid emporium model as one common, overarching 
style of teaching and assumes that the primary change affecting all developmental math 
courses at these colleges from 2006-07 to 2010-11 was the adoption of the hybrid 
emporium model. While the pedagogy of this model is largely consistent, individual 
colleges and faculty members may have adopted additional technologies or offered other 
additional instruction. I was not able to observe, for example, whether faculty brought 
their class together at certain times during a class period to provide additional whole-class 
instruction, or supplemented the Pearson software (which all but one of the colleges used) 
with other technology-centered strategies. There may have been numerous small 
adjustments made to the technology-based course by instructors, and the effects of those 
adjustments on student outcomes are not distinguishable in this study. 


Cost Effectiveness of Technological Interventions 


A primary reason many colleges cite for adopting technology in the classroom is 
lowering the cost of instruction. Computer-based instruction can allow for larger class 
sizes and potentially lower-cost faculty, either through higher faculty course loads or the 
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hiring of adjunct faculty to facilitate the computer-based sections. Students may also save 
money through computer-based courses, potentially through the individualized pacing 
that allows them to complete their courses more quickly, or through saving money on 
textbooks (Twigg, 2013). However, simply reducing costs without improving student 
success will not result in net improvements in outcomes for either the institution or the 
student. Colleges need to evaluate both the benefits and the costs associated with remedial 
redesigns (Belfield, Jenkins, & Lahr, 2016), yet rigorous cost-effectiveness evaluations 
are generally rare in education (Levin & Belfield, 2015). 


Several descriptive cost studies have found that computer-assisted instruction is 
more cost-effective than peer tutoring or traditional instructional formats (Buzhardt, & 
Semb, 2005; Niemiec, Sikorski, & Walberg, 1989). Twigg (2013) summarized the cost 
savings of moving to the adoption of the hybrid emporium model at 32 colleges across 
the country, comparing instructional costs prior to adopting the model to the costs after. 
Thirty-one of the 32 schools in the study reduced costs by an average of 20 percent after 
adopting computer-based instruction. However, the cost savings resulting from the 
adoption of the hybrid emporium model was not compared directly to the outcomes for 
students in this analysis. 


Three technology-based and/or developmental math studies have specifically 
examined the costs of interventions compared to student outcomes. In the high school 
context, Barrow, Markman, and Rouse (2008) conducted a simulation to compare the 
cost-effectiveness of computer-based algebra instruction to a reduction in class size. They 
found that the cost-effectiveness of computer-assisted instruction was comparable to 
reducing class size from 30 to 14 students. Belfield, Jenkins, and Lahr (2016) conducted 
a cost-effectiveness study of corequisite remedial courses in Tennessee. In their analysis, 
they considered both the gain in college-level gateway course pass rates as a result of 
corequisite courses alongside the transition costs of corequisite remediation and the per- 
student costs of the courses. They concluded that the return on investment for corequisite 
courses is high, particularly when considering the impacts on college-level course pass 
rates and subsequent course completion. In a study of online courses across six different 
four-year universities, Bowen, Chingos, Lack, and Nygren (2014) found that there were 
no differences in the learning outcomes for students randomly assigned to a hybrid or 
traditional statistics course. Through simulations they estimated that the cost savings for 
institutions three to five years after launching the online courses were 36 to 57 percent. 
So while there were not positive gains in student outcomes as a result of online courses, 
these courses did reduce costs for the institution. 


In the current study, I find that the adoption of the hybrid emporium model does 
not lead to improved outcomes for students in passing college-level math or in degree 
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completion. Although college-level math completion (at both two- and four-year 
colleges) and degree attainment (at two-year colleges) were lower after the adoption of 
the hybrid emporium model, colleges might still be interested in continuing the model if 
it lowers costs for students or institutions. If the hybrid emporium model lowers the per- 
student cost of developmental math, then perhaps colleges are willing to accept declines 
in college-level math success, particularly if more students enroll in college-level math. 
However, the money colleges might save on instructional costs does not necessarily equal 
the longer term financial loss of not passing college-level math courses or of not earning 
a degree. 


The field of higher education specifically needs more evidence of cost savings 
and cost benefits of interventions and reforms. To build this evidence for the use of the 
hybrid emporium model in developmental math offerings, the cost for each student in a 
hybrid emporium course must be calculated and compared to the per-student cost in a 
traditional course. Many specific elements go into these calculations, including equipment 
costs, faculty and staffing costs, software costs, and changes in class size, to name a few. 
While in some circumstances cost savings are immediately realized, in other cases, it 
might take a longer time for the cost savings to become apparent. The adoption of the 
hybrid emporium model has not yet been rigorously evaluated for cost savings to the 
institution, yet the fact that the results from this study suggest that it is not particularly 
efficacious implies that the return on investment would not be high. This is an important 
area for additional inquiry, as I do not observe gains in student success as a result of the 
hybrid emporium model, and the cost savings are not known. 
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6. Conclusion 


As increasing numbers of colleges and state systems explore the use of 
technology-centered instruction in developmental education, it is important to understand 
the impacts of this shift on students’ course performance and later academic progress. 
There are many reasons why technology could prove effective in teaching basic math 
skills and, conversely, many reasons why it could prove ineffective. Results from this 
study provide evidence that adopting technology-based instruction in developmental 
math courses does not lead to lower pass rates in developmental math itself for college 
students in Tennessee. In fact, at four-year colleges, students in this study passed their 
developmental math courses at higher rates under this model. At both two- and four-year 
institutions, however, students assigned to hybrid emporium developmental math courses 
had lower pass rates in their first college-level math course. While more students enrolled 
in college-level courses, more students also failed those courses. At the community 
colleges, there is evidence that this lower pass rate in college-level math courses may 
have repercussions for credential completion, as students who were assigned to a hybrid 
emporium developmental math course earned associate degrees at lower rates than their 
peers. 


In the years since the adoption of the hybrid emporium model in Tennessee, the 
state has moved toward a credit-bearing, corequisite model for offering developmental 
courses. Since the fall of 2015, all developmental math courses (at both two- and four- 
year colleges) are still taken as a separate course, but now in the same semester as a 
college-level math course. As a result, the alignment in both the content and the 
instructional method has improved across the developmental and college-level course 
transition, with many of the introductory college-level math courses now also using the 
hybrid emporium model. Several colleges now also emphasize the importance of having 
the same instructor in both the developmental (corequisite) course and the college-level 
course. This degree of faculty alignment across developmental and college-level math 
was notably absent in the early years of the adoption of the hybrid emporium model and 
has the potential to reverse the negative impacts on college-level math performance 
observed in this study. For states and systems looking to adopt the technology-based 
instructional model for their developmental math courses, the results of this study speak 
to the importance of considerations regarding alignment in faculty and instructional 
methods across developmental and college-level courses. 
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